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veloped from a history of basic and translational research. This effectiveness trial, with 41 pub- 
lic middle schools randomly assigned to intervention or control, examined student-, teacher-, 
and parent-reported outcomes, as well as math and reading scores and school attendance. Mul- 
tilevel analyses suggested that for students at risk for behavior problems, immediate-interven- 
Keywords: a tion schools outperformed control schools on parent-reported negative school contacts for 
Family management training students at risk for behavior problems. Implementation, however, was hampered by several 
Positive Family Support ; : : i : : 

7 boned . : challenges, including school funding cuts, lack of staff time to provide parenting supports, 
Positive behavior interventions and supports F Pyaar: : Sin rag aha aa 
Effectiveness trial and staff turnover. Given that preventive interventions are generally cost effective, it is critical 
Scale-up that researchers continue their efforts to refine these interventions and find ways to support 
schools’ implementation of evidence-based programs that can reduce problem behavior. This 
article is part of a special issue “Parental Engagement in School-Based Interventions”. 

© 2017 Society for the Study of School Psychology. Published by Elsevier Ltd. All rights 
reserved. 


1. Introduction 


Student problem behavior at school is one of the most stressful challenges facing teachers and school staff (Sugai & Horner, 
2002; Walker, Colvin, & Ramsey, 1995) and one of the most costly problems facing communities and society (Miller, 2004). 
Youths who engage in problem behavior at school often have a variety of related concerns, including low achievement, low school 
attendance, depression, and substance use (Boles, Biglan, & Smolkowski, 2006; Kellam, 1990; Patterson, Reid, & Dishion, 1992). 
The transition to high school is a risky period for youths and is characterized by increased antisocial behavior, including substance 
use and violence, especially for youths with deviant peer relations (Dishion, Capaldi, Spracklen, & Li, 1995; McIntosh, Flannery, 
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Sugai, Braun, & Cochrane, 2008). By ninth grade, having had just one suspension doubles the chance that a student will drop out 
of public school (Balfanz, Byrnes, & Fox, 2013), suggesting the need for services that address problem behavior as early as middle 
school. Many of the behavioral problems that define the risk trajectory for serious delinquency and early-onset substance use are 
most apparent and predictable from prior behavior in the school setting and from disorganized family management practices 
(Dishion & Patterson, 1993; Schaeffer, Petras, lalongo, Poduska, & Kellam, 2003). Thus, early adolescence represents a window 
of opportunity to intervene with students who are currently displaying behavior problems and are prone to escalating these be- 
haviors in adolescence. 


1.1. Family management interventions 


Research supports the value of parental involvement and monitoring as predictors of academic achievement (Spera, 2005) and 
the need for interventions that target parenting practices for high-risk students (e.g., Stormshak, Connell, & Dishion, 2009). Family 
management skills have an influence on the developmental pattern underlying adolescent problem behaviors (Fosco, Dishion, & 
Stormshak, 2012; Patterson & Dishion, 1988; Peterson, Hawkins, Abbott, & Catalano, 1994). Supportive family relationships reduce 
the risk of substance use and later problem behavior (Connell, Dishion, Yasui, & Kavanagh, 2007; Padilla-Walker, Nelson, Madsen, 
& Barry, 2008; Stormshak, Fosco, & Dishion, 2010; Szapocznik et al., 1991). Some reviews of effective interventions to reduce 
youth problem behavior have revealed that family-centered treatment models have the largest effects over time (e.g., Sanders, 
2012; Weisz & Kazdin, 2010; but cf. McCart, Priester, Davies, & Azen, 2006). 

Despite the evidence of benefits, a very small percentage of parents participate in parenting or family interventions to address 
behavior problems (Prinz & Sanders, 2007; Zubrick et al., 1995). Although relatively brief parent interventions in public schools 
may motivate positive change in parenting and reduce problem behavior (Dishion & Kavanagh, 2003; Forgatch, Bullock, & 
DeGarmo, 2003; Kazdin, 2002), few public middle schools integrate empirically supported mental health and parent interventions 
into their behavior management armamentarium (Atkins, Hoagwood, Kutash, & Seidman, 2010). Failure to adopt these practices 
may have been a product of multiple barriers. The costs and logistics of implementation, time needed to train school administra- 
tors or specialists to execute the intervention, inability to secure the requisite professionals for training, inability to reach parents 
and provide services to families, and competing priorities limit the ability of schools to implement interventions that involve fam- 
ilies (Forman, Olin, Hoagwood, Crowe, & Saka, 2009). As such, most interventions that target parenting practices are unrealistic for 
schools (Christenson, 2003). Hence, cost-effective and efficient strategies for engaging parents within school systems that offer a 
clearly defined set of evidence-based behavior management and academic support strategies are warranted. 


1.2. Positive Family Support within tiered behavioral supports in schools 


Multitiered frameworks such as positive behavioral interventions and supports (PBIS; e.g., Anderson & Kincaid, 2005; Horner, 
Sugai, Todd, & Lewis-Palmer, 2005) that embed services within a model of universal, selected, and indicated interventions deliv- 
ered in the school (Horner, Sugai, & Anderson, 2010) provide this context for integrating family-centered interventions. PBIS fo- 
cuses on positive, nonaversive, and systems-change approaches, which have been shown to reduce problem behaviors within 
school settings (e.g., Bradshaw, Mitchell, & Leaf, 2010; Horner et al., 2009; Lane, Wehby, Robertson, & Rogers, 2007; Lassen, 
Steele, & Sailor, 2006; Smolkowski, Strycker, & Ward, 2016; Waasdorp, Bradshaw, & Leaf, 2012; Ward & Gersten, 2013). Many 
schools implement PBIS practices schoolwide, in the classroom, and when working with individual students. Common schoolwide 
PBIS implementations embrace a set of core principles: (a) all stakeholders share the operational set of values, beliefs, vision, mis- 
sion, and purpose that shapes the climate and culture of the school and classrooms; (b) a small set of positively worded behav- 
ioral expectations can be clearly defined and actively taught; (c) appropriate behaviors are reinforced; (d) problem behaviors are 
viewed as an opportunity to teach, with corrections applied calmly and consistently; (e) decisions about students and systems are 
driven by data about office referrals and other aspects of school functioning; and (f) administrators are an active component in 
the process. Horner et al. (2010) discuss schoolwide PBIS as an educational practice and the evidence base for primary 
(schoolwide) interventions, as well as secondary and tertiary interventions. Schools that implement PBIS may involve individual 
parents or adopt specific home-school collaboration practices to support their students (Horner et al., 2010), such as sending 
home a daily behavior report card (Chafouleas, Riley-Tillman, & Sassu, 2006) or Check-In/Check-Out point cards for parents to re- 
view (Turtura, Anderson, & Boyd, 2014). Few schools integrate parent management training or other parenting supports into their 
portfolio of strategies, but features of the tiered behavior support systems may serve as ideal points of entry for more intensive 
parenting and family management services. 

The Positive Family Support (PFS) model is a school-based approach to providing a range of family management interventions 
for middle school youths and their caregivers. To improve the efficiency and uptake of evidence-based family supports, such as 
the Family Check-Up (FCU; Dishion & Stormshak, 2007), the intervention model was redesigned to fit within the PBIS system 
by using principles of systemic concatenation (Dishion, 2011). Systemic concatenation entails adopting the core intervention prin- 
ciples (see concept of kernels; Embry & Biglan, 2008) found to be effective in evidence-based programs into a delivery system that 
improves the workflow and ecology of the school environment. In systemic concatenation with the FCU model, the scope and se- 
quence of family supports align neatly within the context of the PBIS system. Consonant with the precursor Adolescent Transitions 
Program (ATP; Dishion & Andrews, 1995), a key feature of PFS is that it is a multilevel intervention model, assessment driven, and 
tailored to the needs of youths and families (Dishion & Kavanagh, 2000). Fig. 1 provides an overview of the PFS model and its 
integration within the PBIS system. Universal interventions built on the PBIS structures emphasize parents’ awareness of school 
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Fig. 1. An overview of how the Positive Family Support (PFS) model integrates within positive behavior interventions and supports (PBIS). Universal interventions 
built on the PBIS structures emphasize parents’ awareness of school expectations, promote student engagement through the family resource center, and recom- 
mend brief workshops to teachers about increasing parent engagement and teacher-parent communication. The secondary or selected level provides more inten- 
sive supports, such as a protocol for engaging parents in the Check-In/Check-Out intervention and supports to improve attendance and homework completion. The 
tertiary or indicated level of supports includes the Family Check-Up, which comprises a family assessment, feedback, and menu of parenting support services, as 
well as parenting support sessions, parent management training, and community referrals. 


expectations, promote student engagement, and recommend brief workshops to teachers about increasing parent engagement 
and teacher-parent communication. The selected level provides more intensive supports, such as a protocol for engaging parents 
in the Check-In/Check-Out intervention and supports to improve attendance and homework completion. The indicated level of 
supports includes the FCU, which consists of a family assessment, feedback, and menu of relevant and available parenting support 
services as well as parenting support sessions, parent management training, and community referrals. 

This strategy allows PFS implementers to take advantage of systems typical of PBIS, such as behavior support that is among the 
top school improvement goals, behavior support teams that use data for decision making, and administration support for preven- 
tive interventions. Adaptations to PFS, however, were needed to facilitate the school-based delivery of the FCU. The standard FCU 
model included videotaping family interactions, for example, which was removed from the protocol, and parent and teacher as- 
sessment ratings used to provide feedback to parents were shortened. This streamlining was intended to facilitate the completion 
of an FCU with parents in one or two sessions when delivered by school staff, in contrast to the minimum 3 h of contact the FCU 
required in efficacy studies (see Dishion & Stormshak, 2007). Parent management training was also shortened to include four 
modules particularly relevant to school staff, based on focus groups with school staff members: (a) positive behavior support 
in the home to increase academic engagement, (b) limit setting, (c) monitoring one's adolescent, and (d) relationship building 
by negotiating solutions to conflict between parents and adolescents. PFS also included content directly related to school success, 
such as parenting strategies to increase homework completion and attendance support. 


1.3. Positive Family Support conceptual model 


PFS content was based on a number of evidence-based parent management and training models, including the Parent Manage- 
ment Training Model (PMTO; Forgatch, Patterson, & DeGarmo, 2005), ATP, and the Everyday Parenting Curriculum (Dishion, 
Stormshak, & Kavanagh, 2011). The conceptual framework for the PFS model is presented in Fig. 2. As can be seen, the PFS ap- 
proach is hypothesized to improve family management and academic support strategies in the home (e.g., reduced family conflict, 
improved student monitoring, increased parental involvement with school and their child, decreased negative school contact, in- 
creased positive school contact). Several studies support the impact of PFS practices on family management and academic support 
strategies (e.g., Fosco, Van Ryzin, Connell, & Stormshak, 2016; Irvine, Biglan, Smolkowski, Metzler, & Ary, 1999). 
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Fig. 2. Conceptual framework for the Positive Family Support intervention. 


Family management and academic support strategies, in turn, are expected to have a positive impact on student social and 
emotional adjustment (e.g., fewer conduct and emotional problems, reduced experimentation with substances, positive interac- 
tions with peers) and improved student engagement in school (e.g., attendance, homework completion, positive attitudes 
about school). Previous research has demonstrated the efficacy of the combination of primary and secondary family management 
strategies on reductions in marijuana use into early adulthood (Véronneau, Dishion, Connell, & Kavanagh, 2016). Van Ryzin and 
Dishion (2012) found reductions in youth antisocial behavior through high school for a high-risk sample. In a replication of the 
multilevel FCU approach, effects were not found on reductions in student problem behavior in middle school, but complier aver- 
age causal effect (CACE) analyses, which estimate a treatment effect among those families that engaged in the FCU, revealed de- 
creases in substance use initiation, antisocial behavior, deviant peer affiliations, family conflict, and risk behavior (Fosco et al., 
2012; Van Ryzin, Stormshak, & Dishion, 2012). CACE models also suggested that engagers in the FCU had less growth in depres- 
sion and family conflict (Fosco et al., 2016). These social, emotional, and school engagement outcomes were then hypothesized to 
lead to students’ success in high school (e.g., academic achievement and school completion). Another CACE model revealed im- 
provement in school grades and attendance through high school (Stormshak et al., 2009). The results of prior research on the 
FCU suggest that, at least for families engaged in the FCU process, students may improve their behavioral, social, and emotional 
outcomes. 


1.4. An effectiveness trial 


In this study we sought to evaluate the effectiveness of the PFS intervention when implemented by school personnel under 
routine conditions. The study examined the impact of PFS within a cluster randomized controlled trial that randomly allocated 
41 Oregon public middle schools to receive the PFS intervention either immediately (n = 21) or after a delay of 3 to 4 years 
(n = 20). School staff members and administrators received training and coaching about how to prepare a family resource center 
(FRC) in their buildings, how to increase parent-school communication, and how to provide parents with the opportunity to par- 
ticipate in the FCU. In an effectiveness evaluation (Gottfredson et al., 2015) intended to test PFS in real-world settings, school per- 
sonnel implemented PFS interventions with students and their families. 

In addition to examining student outcomes, the study also examined implementation fidelity, processes, and sustainability 
through a multi-stakeholder process evaluation. The blending of effectiveness and implementation research components in this 
way constitutes an effectiveness-implementation hybrid design, as described by Curran, Bauer, Mittman, Pyne, and Stetler 
(2012), and specifically, a Hybrid Type 1 design. The process evaluation, conducted during the rollout of the PFS model, investi- 
gated the barriers and facilitators to implementation to further our understanding of treatment heterogeneity. These implemen- 
tation findings have been presented in previous reports: Fosco et al. (2014) present the history of PFS, its integration within PBIS, 
and details about its components and implementation, including lessons learned through work with the first 21 schools. 
Stormshak et al. (2016) extend the work with a discussion of challenges and barriers encountered taking PFS to scale in middle 
schools, such as limited resources, lack of staff training on key skills, and administrator turnover. Dishion et al. (2016) discuss the 
three primary barriers and potential solutions that use digital technology that may help schools provide tailored and proactive 
parent support. 

As an effectiveness trial, the project required an evaluation by a team external to the PFS developers. The evaluation team con- 
ducted random assignment, collected all data, and conducted statistical tests for this report. The full team, including developers 
and interventionists, chose the measures that best represented the intervention targets that align with the conceptual framework 
presented in Fig. 2. This approach abides by standards of evidence for effectiveness trials in prevention science research 
(Gottfredson et al., 2015). Specifically, we aimed to meet the desirable standard that “a researcher who is neither a current nor 
past member of the program developer's team should conduct data collection and analysis” (Gottfredson et al., 2015, p. 912). 
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Schools assigned to the immediate-intervention condition (hereafter intervention) were compared with schools assigned to the 
delayed-implementation condition (hereafter control) on student- and parent-reported behavior, teacher report of school-based 
family practices, and school archival academic outcomes. The first of two primary research questions asked whether students 
in schools provided with PFS improved on measures of constructs depicted in Fig. 2. Because well-adjusted students and well- 
managed families would not likely require the supports offered through PFS, and because the research base suggested the impor- 
tance of engaging high-risk families, we hypothesized that level of risk would moderate intervention effects; that is, those stu- 
dents with greatest risk are most likely to participate in aspects of PFS (e.g., Stormshak et al., 2009) and be the most likely to 
benefit. This also represented a primary research question. 

We also had several secondary questions. Because we recruited two successive cohorts of students within each school, we test- 
ed for moderation effects by cohort, but had no a priori hypothesis that one cohort would benefit more from PFS than would the 
other. We also tested gender and minority status as moderators, but again we had not hypothesized that the effects of PFS would 
differ by these student characteristics (Barrera, Castro, & Biglan, 1999; Gonzales et al., 2012). 


2. Method 


This study was designed to test the effectiveness of the PFS intervention by using a cluster randomized controlled trial that 
nests students and staff within middle schools that had implemented key features of schoolwide PBIS systems. Investigators 
had planned to recruit 40 to 44 schools that met inclusion criteria and agreed to participate, and then assign half to each of 
two conditions: immediate PFS implementation intervention or delayed implementation control. The design allowed for the de- 
tection of effect sizes (Hedges' g; Hedges, 1981) in the range of 0.30 to 0.39, with a conservative Type | error rate (a) of 0.001 
to account for multiple tests. 


2.1. Study design 


The design matched the original specification, with only minor exceptions. Participating schools were enlisted into the study in 
three annual recruitment waves rather than two as planned. Project staff recruited and randomized 14 schools before the 2009- 
2010 academic year, 13 schools before the 2010-2011 academic year, and 14 schools before the 2011-2012 academic year. We 
randomly assigned schools to immediate or delayed PFS implementation within districts and matched on total enrollment. Single 
schools within a district were matched with other single schools by enrollment, and in the 2010-2011 school year, one school 
was unmatched and assigned at random condition by itself. Schools assigned to the immediate intervention condition began im- 
plementation 1 year after recruitment and randomization. Schools assigned to the delayed control condition began implementa- 
tion 3 years after recruitment and randomization. Fig. 3 depicts the timeline and Fig. 4 presents a school and participant flow 
diagram. 

Within each school, we recruited two successive cohorts of sixth grade students. Cohort 1 entered sixth grade immediately 
after recruitment and randomization, 1 year before PFS implementation in the intervention schools. Cohort 2 began sixth grade 
the next year, 1 year after recruitment and randomization and during the first implementation year in intervention schools. With- 
in intervention schools, all students in Cohort 1 finished sixth grade before implementation, and all students in Cohort 2 began 
sixth grade after implementation. We followed each cohort through eighth grade, which means that students in Cohort 1 received 
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Fig. 3. Positive Family Support study timeline. The timeline depicts the three waves of recruitment and randomization (*g) and the intervention timing for imme- 
diate-intervention (PFS) and delay (Control) schools. Each PSF school participated in assessments only (Assess) in their first year, received training (T) and ongoing 
coaching in the Positive Family Supports model before their second year of participation, and continued to implement PFS and participate in assessments there- 
after. Control schools participated in only assessments during their first 3 years and then received the same training and coaching in the beginning of their fourth 
year. Within each wave of schools, we followed two cohorts of students. Cohort 1 entered sixth grade during the first assessment year and exited middle school in 
the third year of each wave. Cohort 2 entered sixth grade in the second year, after implementation began in PFS schools, and completed eighth grade in the fourth 
year of a schools participation. 
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Fig. 4. Number of schools approached (approximate) for participation and assigned to condition. Each arm indicates the number of schools, teachers, parents, and 
students allocated and who participated by time and recruitment wave. 


2 years of exposure to intervention activities and students in Cohort 2 received 2 years of exposure. The final wave of schools, 
however, did not complete the eighth grade assessment for Cohort 2 because of insufficient time. 


2.2. Participants 


2.2.1, Schools 

Schools with a minimum of 50 students in sixth grade that had implemented schoolwide PBIS systems were eligible for the 
study. We identified approximately 200 schools that met study inclusion criteria and contacted administrators at each of the 
schools with an invitation to participate in the study. After receipt of district approval, researchers made on-site presentations 
and held phone conferences to describe the study to schools that had expressed interest in the study. Schools that agreed to par- 
ticipate in the study signed a memorandum of understanding. 

This effectiveness study included 41 middle schools in 27 school districts in the Pacific Northwest region of the United States. 
The average enrollment of participating schools ranged from 151 to 1037, with five schools considered small (<250 students), 18 
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considered medium (251 to 500 students), and 18 considered large (>500 students). The proportion of students who received 
free or reduced-price lunch ranged from 29% to 94% (median 58%). The proportion of English language learners ranged from 
0% to 37% (median 3%). The proportion of minority students ranged from 5% to 82% (median 28%). The average total Schoolwide 
Evaluation Tool (SET; Horner et al., 2004) score was 81.6 (SD = 13.2), implying that the average school met implementation stan- 
dards for schoolwide PBIS. Schools were found to be comparable across the two study conditions with regard to each of these 
baseline characteristics (all p-values > 0.16). 


2.2.2. Teachers 

Within each school, we recruited teachers to provide reports about their students, contacts with parents, and related information. 
We targeted teachers with the most contact with students (e.g., homeroom teachers). The 653 participating teachers were most often 
male (64%), Caucasian (93%), had a master's degree (69%), and had been teaching an average of 13.8 years (SD = 8.7). Table 1 de- 
scribes the teacher sample by study condition; teachers from both conditions were comparable on all characteristics (all p- 
values > 0.45). 


2.2.3. Students 

As described previously, this study included two cohorts of students, with 6921 in Cohort 1 and 5991 in Cohort 2. To recruit 
students, we used a passive (waiver of) consent process. The parents of 14,331 students received a letter that described the study 
along with a decline postcard they could return if a parent did not want his or her student to participate. The parents of 1419 
students (9.9%) returned the decline card, leaving 12,912 participating students. Students were on average age 11.9 years 
(SD = 1.) at their first report. About half were female (51%) and most were Caucasian (65%). A sizable proportion reported 
their families had “just enough money to get by” (45%). Table 1 describes the student sample by study condition; students 
from both conditions were comparable on all characteristics (all p-values > 0.10), with the exception of report of how much 
money their family had (p < 0.001). 


Table 1 
Teacher and student sample characteristics by treatment condition. 
PFS intervention Delayed control 
Teacher sample characteristics 
Sample, N 337 316 
Female, % 34.7 36.7 
Race or ethnicity,? % 
American Indian or Native American 53. 2.5 
Asian 0.3 1.9 
Black or African American 0.6 0.6 
Hispanic or Latino 24 3.2 
Native Hawaiian or Pacific Islander 0.0 0.3 
White or Caucasian 92.0 93.3 
Highest degree obtained, % 
High school diploma 1.2 2.9 
Associates 5.6 5.9 
Bachelors 22.6 22.8 
Masters 70.3 68.1 
Doctorate or law degree 0.3 0.3 
Years teaching professionally, mean (SD) 13.7 (8.3) 14.0 (9.1) 
Years teaching at this school, mean (SD) 8.0 (6.4) 8.4 (6.2) 
Student sample characteristics 
Sample, N 6457 6455 
Female, % 48.5 49.0 
Race or ethnicity,? % 
American Indian or Native American 19.8 20.2 
Asian 6.2 5:9 
Black or African American 6.0 6.6 
Hispanic or Latino 22.1 26.3 
Native Hawaiian or Pacific Islander 3.0 33 
White or Caucasian 69.8 65.4 
Other 13.6 13.0 
How much money does your family have? % 
Not enough to get by 5.6 5.8 
Just enough to get by 43.9 46.7 
We only have worry about money for fun 34.2 34.0 
We never have to worry about money 16.3 13.7 
Age at first assessment, mean (SD) 12.0 (1.7) 11.9 (0.9) 


* Multiple responses allowed. 
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2.2.4, Parents 
Of the 12,912 students who participated, 5003 parents (38.7%) completed at least one survey. No parent demographic data 
were collected because they were not the focus of our study hypotheses. 


2.3. Intervention 


The PFS school-based intervention was adapted to align with the three-tier model that is based on the systemic concatenation 
strategy described by Dishion (2011). The PFS is tailored according to the level of risk, with graded intervention intensity to ad- 
dress students at the universal, selected, and indicated levels. 


2.3.1. Universal level 

There are three core intervention elements at this level. First, each school established an FRC as a base of operations from 
which trained school personnel can disseminate evidence-based parenting information, such as brochures, books, worksheets, 
and videos. These materials offer parents approaches to problem solving, improving home-school communication, encourage- 
ment, supervision, setting limits, getting to know the friends and peers of their teens, and communication. School personnel 
can also be available to offer basic informational and consultation services to all families of children at the school, including assis- 
tance with accessing online grades or connecting with teachers. Second, family-school partnerships were promoted through par- 
ent outreach activities, parenting topic nights, family activities at the school, and positive family contacts about student successes. 
Third, a schoolwide multiple-gating system was implemented to facilitate early detection of problems and efficient referral to 
more intensive support as needed (Dishion & Patterson, 1993; Loeber, Dishion, & Patterson, 1984). Ideally, this process was ac- 
complished throughout the year with parent school-readiness screening surveys, teacher behavioral screening surveys, and atten- 
dance and disciplinary referral data monitoring. 


2.3.2. Selected level 

At the second tier of the program, schools were trained to implement an enhanced version of the Check-In/Check-Out system 
(Crone, Hawken, & Horner, 2010), which includes a family incentives component to promote student behavioral change at school. 
Traditionally, this system enlists students and teachers to track standardized behavioral goals throughout the day, such as remain- 
ing seated and quiet unless otherwise permitted, and allows students to check out with their behavioral tracking sheets and re- 
ceive rewards for meeting goals. PFS capitalizes on this opportunity to explicitly define parental involvement in supporting 
positive behavioral change and to integrate parents into the early stages of school-based behavioral concerns. Additional supports 
included home-school family management videos and worksheets to foster effective structuring and supervision of homework 
and attendance. The worksheets on homework, for example, provided specific behavioral guidance for designating a standard 
time and place to work and reducing distractions. 


2.3.3. Indicated level 

Interventions at the third tier offered more intensive support for high-risk students or those for whom selected-level supports 
were unsuccessful. Intervention is delivered via the FCU, which we modified to include two brief, family-centered sessions to mo- 
tivate parents to change parenting practices and use intervention services that address their specific needs. The FCU draws on mo- 
tivational interviewing principles (Miller & Rollnick, 2002) to help parents effectively implement family management strategies to 
address student behavioral concerns. During the first session, a parent consultant asked parents about their goals, concerns, and 
motivation for change. Next, parents complete a survey to identify the ecological, family, and youth dimensions that underlie stu- 
dent risk or resilience in the school setting. Based on this information, parent consultants give strengths-based feedback, describ- 
ing the assessment results in a way that supports parent motivation to change and helps identify appropriate evidence-based 
intervention options (e.g., school-based supports for the student, family support programs focusing on parenting skills, communi- 
ty referrals). Families, then, select only those program components they are motivated to engage in. Families can elect to receive 
more intensive family support derived from the Everyday Parenting Curriculum (Dishion et al., 2011), including assistance with 
positive behavior support, parental monitoring, limit setting, and family negotiations. 


2.3.4. Training and technical support 

Because organizational change requires staff motivation (Fixsen, Blase, & Van Dyke, 2011), we conducted workshops with all 
school staff members prior to implementation. Our goals were to provide information about the PFS model, to assess school staff 
needs, and to identify areas of family support the staff were particularly motivated to engage in. To decrease reliance on school 
counselors who had little time to conduct the interventions, we shifted program delivery from a single family resource specialist, 
the method used in prior research, to a range of school staff, including school administrators, instructional and educational assis- 
tants, school receptionists, and teachers. 

The PFS implementation plan included a combination of the school training workshops and technical assistance for school per- 
sonnel. Implementation started with 2-day workshops before the start of the school year. The focus of these workshops included 
the universal and selected levels of the PFS model. Following these workshops, project trainers provided ongoing support and 
technical assistance to ensure ease and efficiency of implementation of the core components at universal and selected levels. 
After the universal-level intervention was in place, school staff received additional workshops about teacher-parent communica- 
tion and training support for implementing selected-level family supports, with schools that were interested and ready to 
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implement beyond universal-level interventions. Next, school staff members were trained to implement the tertiary-level PFS in- 
terventions, including the FCU and follow-up training modules. To encourage staff to use these indicated PFS strategies, an indi- 
vidualized mentorship model was used. As school staff members gained competence conducting FCUs, consultant support was 
gradually decreased and shifted to a consultation model of implementation described next. 

The ongoing technical support strategies consisted of at first, weekly, and at the least, monthly (often dictated by proximity 
and school staff schedules/availability) consultation meetings with school staff. Depending on the particular school, consultation 
meetings consisted of group meetings and participation during which increased family involvement was encouraged and specific 
suggestions were offered, in ongoing PBIS team meetings, team meetings with designated PFS staff, individual consultations with 
counselors or other staff working to implement various aspects of the PFS model, and consultations with administrators about en- 
tire PFS implementation progress. Second, consultants also responded to any school staff who requested support, additional ma- 
terials, or information via e-mail or phone. Third, midyear and end-of-year (during the first and second years of implementation) 
fidelity and progress consultations were conducted with PFS teams or administration. We used these feedback sessions to collab- 
oratively set goals for the coming academic year. 

Consultations were supported by a structured implementation manual (Dishion, Fosco, Moore, Falkenstein, & Stormshak, 
2015). The manual was supported by digital materials available from the University of Oregon Child and Family Center, and 
was given to schools on DVD. Parent engagement materials included template letters to parents that could be adapted to local 
conditions for administrators to use, parent information night materials, PowerPoint presentations for staff discussions with par- 
ents, Spanish translations of parent materials, Excel spreadsheets for sorting and analyzing parent-report screeners, and video 
support materials for positive behavior support in the third tier, including specific strategies for reinforcing homework comple- 
tion, attendance and positive behavior at school, monitoring and setting limits on problem behavior, and negotiating and solving 
conflicts between parents and adolescents. 

Following the initial workshops, which were standard across all schools, individual consultations were tailored to meet the 
specific needs of school staff across the 21 intervention schools. Tailoring of consultation was based on the number of school 
staff members who could allocate time to give parents PFS support and number of school staff members trained in behavioral 
principles. When staffing resources were lean, the project staff members helped school teams develop a feasible PFS strategy 
that focused on increasing the level of universal supports to parents, often involving brief workshops to teachers about strategies 
for parent engagement and proactive, positive communication. 

With respect to Tiers 2 and 3, project trainers provided direct modeling and demonstration of parent engagement in the 
Check-In/Check-Out and FCU processes. Trainers advised school staff to begin Tiers 2 and 3 supports with parents of students 
who had minor behavioral problems, until staff skill levels became fluid and they were able to extend PFS practices with parents 
of students with more significant problem behaviors at school. The greatest challenge to implementation was encouraging staff to 
use Tiers 2 and 3 strategies in the context of middle schools that had scarce resources, and other professional practices were pri- 
oritized over proactive consultations with parents. Often district statutes regarding suspension and expulsion took precedence 
over meetings with parents and other family members. 

To offset implementation costs and secure dedicated staff time during the second year, schools received a total of $10,000 in 
payments split across 4 years, which is similar to the incentives offered by the Oregon Department of Education (ODE) in their 
PBIS initiatives (e.g., Chaparro, Smolkowski, Baker, Hanson, & Ryan-Jackson, 2012). Schools in the control condition conducted 
business as usual, including all practices associated with schoolwide PBS and with individual systems. Control schools received 
access to PFS training systems at the conclusion of the study, including the implementation payments. 


2.4, Procedures 


The appropriate Institutional Review Board approved study procedures. Participating schools provided parent/guardian contact 
information. On behalf of the child's school, parents were mailed a description of the study, two consent forms, and a decline card. 
The first consent form requested child participation in an annual in-class confidential survey and permission for an annual con- 
fidential teacher evaluation of their child. The second consent form asked parents to agree to their own participation in an annual 
confidential mail survey. Parents had the choice to decline their child's or their own participation in the study. Consent forms 
were translated to accommodate non-English-speaking parents (e.g., Spanish, Russian). 

Students whose parents did not decline their participation were administered a survey during the winter term. Prior to admin- 
istration students were given the opportunity to decline participation. Teachers were present during the student survey assess- 
ment, but a study research assistant administered the survey to protect confidentiality. Teachers and parents completed 
surveys each spring. Student and teacher surveys were administered in paper-and-pencil format and via a secure web-based 
data collection program. Parents and teachers received monetary compensation for their participation. Spanish versions of surveys 
were administered when requested. 


2.5. Measures 


Data collection primarily addressed student academic and behavioral outcomes, with measures collected from students, par- 
ents, teachers, the ODE, and fidelity of implementation interviews and direct observation. Consistent with effectiveness trials 
(Flay et al., 2005), implementation measures assessed schools’ support for parents, in general, and exposure to and use of PFS, 
specifically. Thus, we collected data about (a) student grades, attendance, engagement in school, and problem behavior; (b) 
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parent contact with schools, use of family services, family functioning, and parenting behaviors; (c) teacher contact with parents, 
their use of family services in their school, and school climate; and (d) PFS and PBIS implementation fidelity. 


2.5.1. Student report data 

Student report data were obtained from the annual student surveys, and the following eight scale scores were designated as 
primary outcomes: parental monitoring, family conflict, conduct problems, emotional problems, substance use, positive peers, 
school participation, and positive school structure. Basic student demographic characteristics were also collected as part of the an- 
nual surveys, and students’ gender and race were designated as moderators of the primary outcomes. 


2.5.1.1. Parental monitoring. Parental monitoring was assessed with nine items from the expanded version of the Parent Monitoring 
scale (Metzler, Biglan, Ary, & Li, 1998) and seven items from the modified Caretaking and Family Routines scale (Metzler et al., 
1998). Students were asked how often at least one of their parents monitored their daily behavior (e.g., who you hang out 
with during your free time). Response items were on a 4-point scale ranging from 1 (never or almost never) to 4 (always or almost 
always), and a mean score was computed (sample a = 0.91). 


2.5.1.2. Family conflict. Family conflict was assessed with a four-item Family Conflict scale (Stormshak et al., 2009) that asked stu- 
dents how many times in the past month a conflict situation had occurred between family members (e.g., we got angry at each 
other). Students responded with a 7-point scale ranging from 1 (never) to 7 (8 or more times), and we computed the mean score 
(sample a = 0.78). 


2.5.1.3. Conduct problems and emotional problems. Conduct problems and emotional problems were assessed with five items from 
the Strength and Difficulties questionnaire (Goodman, 2001). Students were asked how true statements were about misconduct 
in the past 6 months (e.g., | get very angry and often lose my temper) and emotional symptoms (e.g., I worry a lot). Response 
options were on a 3-point scale ranging from 1 (not true) to 3 (certainly true), with a mean score computed for conduct problems 
(sample a = 0.63) and emotional problems (sample a = 0.69). 


2.5.1.4. Substance use. Substance use was assessed with three items from the Problem with Students scale (Metzler, Biglan, Rusby, 
& Sprague, 2001). Students were asked how often they used tobacco, alcohol, and marijuana or other drugs. Response options 
were on a 5-point scale ranging from 1 (not at all) to 5 (very often), with a mean score computed for analysis (sample a = 0.87). 


2.5.1.5. Positive peers. Positive peers measures included three items modified from the Child Peer Social Skills questionnaire 
(Dishion & Kavanagh, 2003) that asked students how many of the students they know had engaged in positive school behaviors 
(e.g., take school seriously). Students responded on a 5-point scale ranging from 1 (very few) to 5 (almost all), and we calculated a 
mean score (sample & = 0.77). 


2.5.1.6. School participation. Measures of school participation included four questions that asked if students liked going to school, 
completed assignments and homework on time, missed or arrived late for school (reversed), and got along with teachers and 
staff. Students responded on a 5-point scale in which 1 = not at all, 3 = sometimes, and 5 = very often. The mean had score re- 
liability of « = 0.56. 


2.5.1.7. Positive school structure. Positive school structure was assessed with 13 items that asked about rule clarity, school response 
to breaking rules, requests to work on special projects, chance to participate in discussions or activities, feelings of safety, recog- 
nition for doing a good job, and similar content. Students responded using a 5-point Likert scale ranging from 1 (strongly agree) to 
5 (strongly agree), and we analyzed the mean (sample a = 0.86). 


2.5.2. Parent report 

Parent report data were obtained from annual parent surveys, and the following seven scale scores were designated as prima- 
ry outcomes: school success, student risk, parental monitoring, parent involvement, negative school contact, positive school con- 
tact, and school structures. 


2.5.2.1. School success. School success was assessed with eight items from the Secondary School Readiness Inventory, designed as 
part of the current study. Items included aspects of behavior (e.g., my child behaves well at school), academics (e.g., my child gets 
grades that are appropriate for his or her skills), attention related to school success (e.g., my child stays on task), and late or ab- 
sent from school. Parents were asked to rate each item on a 10-point scale ranging from 1 (an area that needs support) to 10 (an 
area of strength). A mean score was computed (sample a = 0.90). 


2.5.2.2. Student risk. Student risk was assessed with six items from a modified version of the Teacher Risk Assessment (TRISK; 
Soberman, 1994). Parents were asked to rate their level of concern for their child using a 4-point scale ranging from 1 (no con- 
cern) to 4 (serious concern) in areas of internalizing behavior (depressed, anxious, or irritable) and externalizing behavior (spends 
time with students who break school rules), attention (able to focus and stay on task in class), sociability (relations with other 
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students), and adherence to rules (follows classroom rules; completes homework and assignments on time). A mean score was 
computed (sample a = 0.82). 


2.5.2.3. Parental monitoring. Parental monitoring was assessed with six items from the modified version of the Caretaking and Rou- 
tines scale (Metzler et al., 1998). Parents were asked how often in the past month they had engaged in monitoring behavior (e.g., 
check in with your child after school about his/her day). Response options were on a 3-point scale ranging from 0 (not at all) to 3 
(very often), with a mean score computed for analysis (sample a = 76). 


2.5.2.4. Parent involvement. Parent involvement was assessed with six items from the Parent Teacher Involvement Questionnaire— 
Parent version (Conduct Problems Prevention Research Group, 1991). Parents were asked how often they interacted with their 
child's teacher or school (e.g., called your child's teacher; attended a special event at your child's school). Response options 
were listed on a 4-point scale ranging from 0 (never) to 3 (weekly or more), and we computed a mean score for analysis (sample 
a = 0.66). 


2.5.2.5. Negative and positive school contact. Negative school contact and positive school contact were each assessed with one item 
designed to tap into targeted areas of the intervention. Parents were asked how often someone from the school had contacted 
them about their child's negative or positive behavior. Response options were listed on a 4-point scale ranging from 0 (never) 
to 3 (weekly or more). 


2.5.2.6. School structures. School structures were assessed with three items designed for our study. Parents were asked to what ex- 
tent they agreed with statements that their child's school had structures in place so they could easily monitor their child's home- 
work, attendance, and behavior. Response options were listed on a 4-point scale ranging from 0 (strongly disagree) to 3 (strongly 
agree), and a mean score was computed (sample a = 0.77). 


2.5.3. Teacher report 
Teacher report data were obtained from annual teacher surveys that asked teachers for their opinions about their school and 
to report about individual student-level characteristics. 


2.5.3.1. School readiness. School readiness was a primary study outcome and a scale was developed for the purposes of this study. 
The four-item scale asked teachers how much they agreed with school-level statements specific to PFS: my school (a) is a “family 
friendly” school or (b) has an effective approach for working with parents to manage student behavior; parents are (c) contacted 
before child behavior problems get out of hand or (d) regularly informed about their student's positive behaviors. Response op- 
tions were listed on a 6-point scale ranging from 1 (strongly disagree) to 6 (strongly agree), and a mean score was computed (sam- 
ple a = 0.78). 


2.5.3.2. Student risk. Student risk was assessed with the six-item TRISK (sample a = 0.89), which is the same scale used with par- 
ents and described previously under “parent report” measures. This measure was designated as a moderator of the primary study 
outcomes. 


2.5.4, State data 

State data for individual students were obtained from the ODE each academic year of the study and three scores designated as 
primary outcomes: end-of-year math scores, end-of-year reading scores, and number of days absent from school. The math and 
reading raw scores were converted to a scale score called a Rasch unit or RIT score (ODE, 2012). 


2.5.5. Fidelity of implementation 
Fidelity of implementation data were collected in the spring of each academic year by independent trained raters on two mea- 
sures: a PFS Fidelity instrument and the Schoolwide Evaluation Tool. 


2.5.5.1. PFS implementation fidelity. The PFS Fidelity evaluation tool is a 33-item assessment developed for this study as an overall 
indicator of school readiness and implementation of key components of the PFS intervention. All schools were assessed with this 
fidelity measure. Raters interviewed the school principal or vice principal and, if available, members of the positive behavior sup- 
port team. Twenty-two items were used to assess universal components of the intervention (e.g., does your school use a system 
for identifying students who are struggling behaviorally), 22 items were used to assess selected or indicated components of the 
intervention (e.g., are parents contacted when students are identified as struggling behaviorally), and 11 items were used to as- 
sess the presence of suggested resources to include in the FRC (e.g., phone to make calls for community resources). Response op- 
tions included a “yes or no” format (yes = 1, no = O) and four category response options (1 = never, 4 = always). Four category 
response options were recoded as “O” if a component of the intervention was never in place and “1” if partially or fully in place. 
Three fidelity scale scores were computed by summing responses to universal, selected, or indicated items, and resource items, 
thus reflecting the number of components of the intervention partially or fully implemented. Fidelity scores showed acceptable 
to excellent score reliability (a) and interrater reliability (intraclass correlation, ICC): universal (sample a = 0.88, ICC = 0.98), 
selected or indicated (sample a = 0.65, ICC = 0.99), and number of resources (sample a = 0.94, ICC = 0.93). 
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2.5.5.2. Schoolwide evaluation tool. The SET is a research-validated instrument designed to assess and evaluate the critical features of 
schoolwide effective behavior support across a school (Todd et al., 2012). The SET includes 28 questions across seven features, 
including (a) expectations defined, (b) behavioral expectations taught, (c) acknowledgment procedures, (d) correction proce- 
dures, (e) monitoring and evaluation, (f) management, and (g) district-level support. Responses were gathered through a review 
of school records, direct observations, and staff and student interviews. Horner et al. (2004) documented score reliability (a) of 
0.96, test-retest agreement at 97%, interrater agreement at 99%, and construct validity correlation with scores from the Effective 
Behavior Support Self-Assessment Survey of 0.75 (see also Vincent, Spaulding, & Tobin, 2010). For our study, interrater reliability 
was excellent, ICC > 0.99. 


2.6. Statistical analysis 


We assessed implementation fidelity after the third year for each recruitment wave with analysis of covariance (ANCOVA) 
models with study condition as a two-level predictor and baseline fidelity scores as a continuous covariate. We assessed interven- 
tion effects on each of the primary outcomes with a random coefficients analysis (RCA; Murray, 1998) or growth model with stu- 
dents, parents, and teachers nested within schools to account for the intraclass correlation. The analysis tests for condition 
differences on growth in outcomes from T1 to T4. The basic statistical model includes time, condition, and the Time x Condition 
interaction, with linear time coded 0, 1, 2, and 3 at T1, T2, T3, and T4, respectively, and condition coded 0 for control and 1 for 
intervention. With 41 schools, tests of Time x Condition used 39 degrees of freedom (df). 

The intervention effects' models were expanded to include moderators to test for differential response to the intervention. To 
test moderation, we expanded the model to include a predictor, its interaction with condition and time, and the three-way inter- 
action between the predictor, time, and condition. The three-way interaction provides an estimate of whether the condition effect 
varied by the predictor. The analysis included dichotomous and continuous predictors, and we used continuous variables when- 
ever possible. 


2.6.1. Model estimation 

We fit intervention effects models to our data with SAS PROC MIXED version 9.2 (SAS Institute, 2009) using restricted max- 
imum likelihood (ML) and included all available data, whether or not scores were present at all four time points following the 
recommendations of Allison (2012). ML estimation with all available data produces potentially unbiased results even in the 
face of substantial missingness, provided the missing data were either missing completely at random or missing at random, mean- 
ing that missing data likely did not depend on unobserved determinants of the outcomes of interest (Graham, 2009; Little & 
Rubin, 2002; Schafer & Graham, 2002). In our study, most missing data were missing by design and therefore missing completely 
at random and ignorable. Students in Cohort 1 within each school provided data from T1 through T3, and Cohort 2 provided data 
from T2 to T4. Most other missing data represented absences on assessment days (e.g,, illness, family vacations, truancy) or stu- 
dents who transferred to a new school between years. Most project schools, however, were in rural communities, so transfers re- 
quired moves to a new town. The transfers were more likely associated with parental decisions, such as employment change, 
financial challenges, or changes in family configuration, and hence balanced across conditions. We therefore believe that most 
missing data, although not missing completely at random, did not represent a meaningful departure from the missing at random 
assumption. Nonrandom missingness was nonetheless possible, but Graham (2009) notes that “[missingness not at random] alone 
is often not sufficient to affect the internal validity of an experimental study to any practical extent” (p. 568; see also Collins, 
Schafer, & Kam, 2001). 

The models assume independent and normally distributed observations. We addressed the first assumption (van Belle, 2008) 
by explicitly modeling the multilevel nature of the data. Regression methods have been found quite robust to violations of nor- 
mality, and outliers have a limited influence on the results in a variety of multilevel modeling scenarios (Bloom, Bos, & Lee, 
1999; Donner & Klar, 1996; Fitzmaurice, Laird, & Ware, 2004; Hannan & Murray, 1996). Murray et al. (2006) showed that viola- 
tions of normality at either or both the individual and group levels do not bias results as long as the study is reasonably balanced 
at the group level. 


2.6.2. Multiple tests 

As noted previously, the project was designed to detect effect sizes (Hedges' g) in the range of 0.30 to 0.39 with a Type | error 
rate (a) of 0.001 and Type II error rate (8) of 0.20. In school-based, cluster randomized trials, the tradeoff between Type I and II 
error rates represents a delicate balance. The cost of a false positive conclusion, a Type I error, such as the assumption that PFS 
improved student behavior in school when it in fact did not, can lead to false expectations about improvements in student behav- 
ior and increased costs associated with implementation of ineffective programs. Conversely, false negatives, or Type II errors, can 
obscure the value of a potentially effective curriculum. Both types of errors can mislead. 

To balance concerns about Type I and Type II errors, we recommend two interpretations of the results. First, for patterns of 
results, we describe the likely number of Type I errors. With 19 primary outcomes and the usual criterion a of 0.05, we anticipat- 
ed a 62% chance of one or more Type I error, a 25% chance of two or more, and a 7% chance of three or more Type | errors. We 
therefore interpret patterns of results for multiple items. Alternatively, we expect zero to three errors with 95% confidence. Sec- 
ond, for the interpretation of an individual test by itself, we recommend a criterion « of 0.003. For the exploration of differential 
effects, we tested four moderators for each of the 19 outcomes. For these 76 tests, we expect between one and eight Type I errors 
due to chance with 95% confidence. For example, there was a 53% chance of four or more Type | errors and an 18% chance of six 
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or more errors. For the interpretation of an individual test, we recommend a criterion of 0.001. To allow for both interpretations, 
we describe results for all tests that were statistically significant using an unadjusted p-value. 


2.6.3. Effect sizes 

To ease interpretation, we computed Hedges' g (Hedges, 1981) for each intervention fixed effect according to the What Works 
Clearinghouse (2014) standards. Hedges' g is comparable to Cohen's d (Cohen, 1988). Both represent individual-level effect sizes, 
but we suggest caution during interpretation, as this study is designed for inferences about schools. An interpretation of school- 
level effects at the individual-student level may be an instance of the ecological fallacy (Snijders & Bosker, 2012). 


3. Results 


Tables 2, 3, and 4 describe the student-reported, parent-reported, and academic measures. The following sections address 
schoolwide PBIS implementation, PFS implementation fidelity, attrition and baseline equivalence, tests of efficacy, differential re- 
sponse to the PFS intervention, and exploratory analyses. 


3.1. Schoolwide PBIS context 


We report schoolwide PBIS fidelity via SET scores to set the context for the results, given that all schools reported 
implementing, at a minimum, the universal, schoolwide tier of PBIS in order to qualify for the study. Schools averaged an 81.6 
on the SET, with 25 of the 41 schools passing the “80/80” criterion for full implementation: 80 or better overall and 80 or better 
on for behavioral expectations taught (Vincent et al., 2010). Neither metric was different, statistically, between conditions. Among 
PFS intervention schools, subscale score means were (a) 75 for expectations defined (range: 25 to 100), (b) 89 for behavioral ex- 
pectations taught (60 to 100), (c) 87 for acknowledgment procedures (50 to 100), (d) 65 for correction procedures (25 to 88), (e) 
94 for monitoring and evaluation (63 to 100), (f) 79 for management (31 to 100), and (g) 79 for district-level support (0 to 100). 
Many schools had fully implemented schoolwide PBIS, but several were missing key features, notably, expectations defined and 
posted, correction procedures (e.g., consequences for behavior problems with staff agreement), and to a lesser extent, manage- 
ment (e.g., behavior team membership, regular meetings, use of action plans). 


Table 2 
Descriptive statistics for student report dependent variables at Year 1 (T1) to Year 4 (T4). 
Positive Family Support Control Percentiles for full sample 

M SD N M SD N Min 25th 50th 75th Max 

Parental monitoring Tl 3.07 0.67 2520 3.08 0.66 2414 1.00 2.62 3.15 3.62 4.00 
T2 3.05 0.67 4844 3.03 0.67 4612 1.00 2.62 3.08 3.58 4.00 

T3 2.98 0.70 4481 2.95 0.69 4481 1.00 2.46 3.00 3.54 4.00 

T4 2.96 0.70 1274 2.88 0.71 1343 1.00 2.46 3.00 3.46 4.00 

Family conflict Tl 2.58 1.40 2542 2.58 1.40 2444 1.00 1.50 2.25. 3.50 7.00 
T2 2.59 1.37 5006 257 1.40 A715 1.00 1.50 225 3.50 7.00 

T3 2.57 133 4561 2.66 1.38 4553 1.00 1.50 2.25 3.50 7.00 

T4 2.45 1.30 1313 2.56 1.35 1378 1.00 1.50 2.25 3.25 7.00 

Conduct problems Tl 1.38 0.37 2429 1.40 0.37 2348 1.00 1.00 1.40 1.60 3.00 
T2 1.37 0.36 4653 1.37 0.36 4522 1.00 1.00 1.20 1.60 3.00 

T3 1.37 0.37 4331 1.38 0.37 4401 1.00 1.00 1.20 1.60 3.00 

T4 1.33 0.36 1241 1.38 0.36 1322 1.00 1.00 1.20 1.60 3.00 

Emotional problems Tl 1.49 0.42 2434 1.49 0.42 2346 1.00 1.20 1.40 1.80 3.00 
T2 1.51 0.43 4659 1.50 0.42 4532 1.00 1.20 1.40 1.80 3.00 

T3 1.54 0.45 4337 1.55 0.45 4409 1.00 1.20 1.40 1.80 3.00 

T4 1.53 0.45 1242 1.58 0.46 1320 1.00 1.20 1.40 1.80 3.00 

Substance use Tl 1.04 0.24 2573 1.05 0.31 2435 1.00 1.00 1.00 1.00 5.00 
se) 1.06 0.35 5097 1.06 0.36 4770 1.00 1.00 1.00 1.00 5.00 

T3 1.11 0.47 4594 1.12 0.50 4594 1.00 1.00 1.00 1.00 5.00 

T4 1.12 0.49 1330 1.11 0.45 1385 1.00 1.00 1.00 1.00 5.00 

Positive peers Tl 3.17 1.02 2497 3.22 1.03 2396 1.00 2.33 3.33 4.00 5.00 
T2 3.21 1.00 4783 3.19 0.99 4555 1.00 2.33 3.33 4.00 5.00 

T3 3.19 1.02 4415 3.15 0.99 4431 1.00 2.33 3.33 4.00 5.00 

T4 3.28 1.00 1254 3.11 0.98 1325 1.00 2.33 3.33 4.00 5.00 

School participation Tl 3.93 0.70 2595 3.96 0.70 2497 1.00 3.50 4.00 4.50 5.00 
T2 3.94 0.66 5133 3.96 0.65 4809 1.00 3.50 4.00 4.50 5.00 

T3 3.88 0.68 4622 3.90 0.66 4634 1.00 3.50 4.00 433 5.00 

T4 3.91 0.66 1336 3.86 0.65 1390 1.00 3.50 4.00 4.25 5.00 

Positive school structure Tl 2.34 0.60 2607 2.29 0.62 2506 1.00 1.92 2.23 2.62 5.00 
T2 2.39 0.64 5122 237 0.62 4804 1.00 1.92 2.31 207 5.00 

T3 2.57 0.67 4617 2.54 0.63 4630 1.00 2.15 2.50 3.00 5.00 

T4 2.54 0.64 1333 2.62 0.61 1390 1.00 2.15 2.54 2.92 5.00 
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Table 3 
Descriptive statistics for parent report dependent variables at Year 1 (T1) to Year 4 (T4). 
Positive Family Support Control Percentiles for full sample 
M SD N M SD N Min 25th 50th 75th Max 
School success Tl 7.74 Lay 1282 es 1.81 1257 1.00 6.75 8.25 9.13 10.00 
72 7.71 1.84 1666 7.74 1.83 1484 1.00 6.75 8.25 9.13 10.00 
T3 7.80 1.85 979 7.81 1.82 981 1.00 6.88 8.27 9.25 10.00 
T4 7.98 1.74 384 7.86 1.81 256 1.00 7.13 8.38 9.13 20.00 
Student risk Tl 1.68 0.64 1280 1.67 0.66 1257 1.00 1.17 1.50 2.00 4.00 
T2 1.66 0.63 1665 1.70 0.65 1484 1.00 1.17 1.50 2.00 4.00 
T3 1.63 0.60 978 1.65 0.62 981 1.00 1.17 1.50 2.00 4.00 
T4 1.58 0.53 384 1.70 0.67 256 1.00 1.17 1.50 2.00 4.00 
Parental monitoring Tl 2,53 0.45 1279 2.49 0.48 1259 0.00 2.33 2.67 2.83 3.00 
T2 2.48 0.47 1659 2.43 0.52 1480 0.00 2.17 2.50 2.83 3.00 
T3 0.38 0.50 975 Zor 0.52 O77 0.00 2.17 2.50 2.83 3.00 
T4 2.40 0.50 385 2.40 0.47 256 0.00 2.17 2.50 2.83 3.00 
Parent involvement Tl 1.13 0.47 1283 1.09 0.46 1255 0.00 0.83 1.00 133 3.00 
T2 1.13 0.47 1654 1.13 0.48 1481 0.00 0.83 1.00 133 3.00 
T3 1.11 0.45 976 1.07 0.46 976 0.00 0.83 1.00 1.33 3.00 
T4 1.11 0.46 382 1.11 0.54 252 0.00 0.83 1.00 1.33 3.00 
Negative school contact Tl 0.42 0.70 1276 0.37 0.64 1242 0.00 0.00 0.00 1.00 3.00 
T2 0.42 0.67 1651 0.42 0.68 1469 0.00 0.00 0.00 1.00 3.00 
T3 0.39 0.63 973 0.34 0.60 973 0.00 0.00 0.00 1.00 3.00 
T4 0.33 0.56 381 0.42 0.63 248 0.00 0.00 0.00 1.00 3.00 
Positive school contact Tl 0.71 0.70 1276 0.71 0.74 1242 0.00 0.00 1.00 1.00 3.00 
T2 0.73 0.70 1651 0.70 0.71 1472 0.00 0.00 1.00 1.00 3.00 
T3 0.68 0.65 972 0.68 0.69 969 0.00 0.00 1.00 1.00 3.00 
T4 0.66 0.67 380 0.82 0.76 250 0.00 0.00 1.00 1.00 3.00 
Positive school structures Tl 2.24 0.67 1272 2.23 0.69 1244 0.00 2.00 2:33 2.67 3.00 
72 2.22 0.69 1648 2.18 0.70 1474 0.00 2.00 2.33 2.67 3.00 
T3 2.21 0.68 973 2.20 0.68 968 0.00 2.00 2.33 2.67 3.00 
T4 2.16 0.66 381 2.10 0.69 250 0.00 1.67 2.00 2.67 3.00 


3.2. PFS implementation fidelity 


We examined differences between conditions in PFS fidelity scores from the second year after implementation with baseline as 
a pretest covariate to determine if the PFS intervention had been delivered as expected. All schools were assessed on PFS fidelity. 
Schools that implemented PFS had significantly greater scores for universal, F(1, 40) = 17.34, p < 0.001, g = 1.07, selected or in- 
dicated, F(1, 40) = 6.55, p = 0.015, g = 0.32, and number of resources available to parents, F(1, 40) = 10.88, p = 0.002, g = 
1.65. Posttest adjusted means showed the significant and large condition effects favored PFS schools for each fidelity score: uni- 
versal (17.72 vs. 12.74), selected or indicated (20.11 vs. 18.88), and number of resources (7.92 vs. 3.73). 


peaks Statistics for Oregon Department of Education (ODE) and teacher report variables at Year 1 (T1) to Year 4 (T4). 
Positive Family Support Control Percentiles for full sample 
M SD N M SD N Min 25th 50th 75th Max 
Math score (ODE) T1 = =224.1 15.6 2526 =. 223.8 13.8 2322 88.0 218.0 225.0 232.0 265.0 


T2 228.6 13.4 4932 228.3 13:1 4426 91.0 222.0 230.0 236.0 279.0 
T3 2335 13:7 4459 233.1 13.6 4289 95.0 226.0 234.0 240.0 285.0 
T4 2348 13.1 1302 235.7 12.2 1366 88.0 228.0 236.0 242.0 279.0 
Reading score (ODE) Tl 225.2 13.9 2516 224.5 13.4 2316 93.0 219.0 226.0 232.0 261.0 
T2 228.4 133 4915 228.5 12.0 4416 98.0 223.0 229.0 236.0 274.0 
T3 232.1 12.4 4427 231.8 11.9 4283 99.0 227.0 233.0 239.0 277.0 
T4 233.4 10.1 1300 =. 232.5 10.4 1275 117.0 227.0 234.0 239.0 271.0 


Days absent (ODE) Tl 8.50 7.72 2554 8.04 7.35 2339 0.00 3.00 6.50 11.50 77.00 
T2 9.39 8.60 5020 8.43 7.95 4546 0.00 3.00 6.50 12.00 100.0 
T3 8.91 8.99 4516 8.64 8.67 4462 0.00 2.50 6.50 12.00 82.00 
T4 9.08 8.61 1312 8.61 8.80 1374 0.00 3.00 6.50 12.00 71.00 

School readiness (teacher report) T1 4.38 0.84 109 4.18 0.98 100 1.25 3.75 4.25 5.00 6.00 
T2 434 0.83 151 4.37 0.85 170 1.25 3.75 4.50 5.00 6.00 
T3 4.09 0.93 107 4.35 0.85 117 1.50 3.50 4.25 4.88 6.00 
T4 4.07 1.02 41 421 1.02 40 195 3.50 425 5.00 6.00 
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3.3. Attrition and baseline equivalence 


Attrition was defined as the 5133 students with preintervention data (i.e., Cohort 1 students assessed at T1) who were missing 
data at either T2 or T3. We did not examine attrition at T4 because most missingness was by design. We experienced 42% cumu- 
lative attrition by T3, with 1075 students missing T2 or T3 data in comparison schools and 1076 missing T2 or T3 data in inter- 
vention schools. Attrition rates did not differ between conditions (y” = 1.42, df = 1, p = 0.232). 

Although differential rates of attrition are undesirable, differential scores by condition present a greater threat to validity 
(Barry, 2005). We conducted an analysis to test whether student scores were differentially affected by attrition across conditions. 
We examined the effects of condition, attrition status, and the interaction between the two on pretest scores within a mixed- 
model analysis of variance (Murray, 1998), which nests T1 scores within schools and condition. We tested all T1 outcomes and 
found no evidence of differential attrition for any of our dependent variables: p > 0.122 for all tests. 

We tested the difference between conditions at baseline within the models that test efficacy. The condition effect (not crossed 
by time) shows the difference between conditions at pretest (see Tables 5, 6, and 7). This difference was not statistically signif- 
icant for any of the student-, parent-, teacher-, or state-reported data. 


3.4, Effectiveness 


3.4.1. Main effects 

To address the first of our primary research questions about overall treatment effects, we tested whether students in interven- 
tion schools would perform better on social, behavioral, and academic measures than would students in comparison schools. 
Tables 5, 6, and 7 present the results of all tests of main-effect tests of treatment efficacy for student, parent, and state and teacher 
reports, respectively. The Time x Condition row represents the critical test of condition on growth for each measure from T1 to 
T4, and the bottom two rows in each table show the effect sizes and p-values for that critical test. Students in schools that imple- 
mented PFS appeared to outperform students in comparison schools on student report of positive peers (g = 0.10), although the 
test did not meet our criterion a given multiple tests. Most importantly, the slope for the control condition was — 0.045 and the 
slope for PFS schools was 0.014 or 0.059 more than the control slope. Table 5 presents the full model. No other significant Time 
x Condition effects were detected. 


3.4.2. Differential effects 

Our second primary research question asked whether students’ behavioral risk moderated intervention effects. To address the 
secondary research questions, we tested whether gender, minority status, and student cohort moderated differences in student 
social, behavioral, and academic outcomes across intervention conditions. In the following paragraphs we present only the statis- 
tically significant moderation effects. 


Table 5 
Results from mixed-model Time x Condition analysis of condition effects on student-reported outcomes. 


Effect or statistic Parental Family Conduct Emotional Substance Positive School Positive school 
monitoring conflict problems problems use peers participation structure 
Fixed Intercept Boo 3.0965""** ~—-2,5803**** 1.3862°°"" 1.4736°°"" = 1.0307°*"" —3.2195°*** ss 3.9884**** 223390" 
effects 0.0263) 0.0401) (0.0131 (0.0102 0.0075) (0.0529) 0.0308) (0.0250 
Condition Bor —0.0125 0.0429 — 0.0056 0.0062 —0.0056 — 0.0767 —0.0419 0.0424 
0.0369) 0.0563) (0.0184 (0.0143 0.0106) (0.0742) 0.0431) (0.0351 
Time Bo2 —0.0731°"** 0.0431" 0.0036 0.0429"""* ~—-0,.0441""**  — 0.0446" —0.0614"*** 0.1509**** 
0.0010) 0.0184) (0.0050 (0.0068 0.0068) (0.0188) 0.0112) (0.0148 
Time x Condition fo3 (0.0126 — 0.0463 — 0.0033 —0.0058 0.0025 0.0588" 0.0114 —0.0085 
0.0138) 0.0258) (0.0070 (0.0010 0.0095) (0.0264) 0.0166) (0.0208 
Variances School intercept 739 0.0114" 0,0218°** 0.0027*** 0.0011** 0.0005* 0.0501°°** 0.0164**** ~—-0.0103*** 
0.0030) 0.0069) (0.0008 (0.0004 0.0002) (0.0125) 0.0042) (0.0030 
School slope 731 0.0018** 0.0037** 0.0003** 0.0006** 0.0005** 0.0052*** 0.0020*** 0.0037*** 
0.0004) 0.0014) (0.0001 (0.0002 0.0002) (0.0016) 0.0006) (0.0010 
Student intercept T29 0.2148°**" ~—-0.8953*"** —-0,0597"*** —-0.0823"*** ——-0,0006 0.3952°*** ——0.2121°"*™* 0.1750°*** 
0.0050) 0.0203) (0.0014 (0.0021 0.0012) (0.0105) 0.0048) (0.0041 
Student slope T21 0.0153"""" ~—-0,0161"* 0.0027°"** ~—-0,0077""** ~—-0.0279"*** ~—s-0,0124**** —s-0.0076**** 0.0118"""* 
0.0014) 0.0055) (0.0004 (0.0006 0.0008) (0.0030) 0.0013) (0.0012 
Residual o 0.2018°""* 0,9486"*** ~—-0.0656"**™* ~—so.osg6"**™* = 0.1015"**™* —0.5426"°** —0.2115°*** 0.1887"""* 
0.0029) 0.0128) (0.0009 (0.0013 0.0013) (0.0073) 0.0029) (0.0026 
Hedges'g Time x Condition g 0.036 —0.074 —0.043 —0.024 0.004 0.101 —0.012 0.027 
p-Values_ Time x Condition p 0.368 0.081 0.647 0.550 0.789 0.031 0.496 0.233 


Note. Table entries show parameter estimates with standard errors in parentheses except for intraclass correlations (ICCs), Hedges’ g values, and p-values. Tests of 
fixed effects (first four rows) used 39 df to account for the school as the unit of analysis. ICC = intraclass correlation coefficient. ICCs calculated as proportion of 
variance in change over time due to students within the same school (731 / [T21 + T31]). 

* p<0.05. 

* p< 0.01. 
*** pb < 0.001. 
*e* p< 0.0001. 
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Table 6 
Results from mixed-model Time x Condition analysis of condition effects on parent-reported outcomes. 
Effect or statistic Parental Parent Student risk School Negative school Positive school School 
monitoring involvement success contact contact structures 
Fixed effects Intercept Boo 2.4962**** 1.7201°°** 7.610°°** 1.1427" 0.4328""** 0.7259**** 22115" 
(0.0236 0.0245) (0.0685 0.0245) 0.0291) 0.0303) 0.0314) 
Condition Bor 0.0344 —0.0151 0.0373 0.0122 0.0191 0.0276 0.0277 
(0.0333 0.0344) (0.0963 0.0345) 0.0408) 0.0426) 0.0441) 
Time Bo2 —0.0629"*** —0.0076 0.0203 —0.0118 —0.0114 —0.0011 —0.0185 
(0.0079 0.0101) (0.0237) 0.0077) 0.0113) 0.0141) 0.0133) 
Time x Condition Po; —0.0029 — 0.0028 — 0.0413 — 0.0036 —0.0011 —0.0180 —0.0003 
(0.0110 0.0141) (0.0329 0.0106) 0.0158) 0.0197) 0.0185) 
Variances School intercept 730 0.0080*** 0.0067** 0.0535"" 0.0091*** 0.0109** 0.0115" 0.0133" 
(0.0023 0.0023) (0.0196 0.0026) 0.0039) 0.0038) 0.0041) 
School slope 731 0.0002 0.0006 0.0019 0.0002 0.0007 0.0011 0.0010 
(0.0002 0.0004) (0.0024) 0.0003) 0.0005) 0.0008) 0.0007) 
Student intercept T20 0.1178"*** 0.2743°*** 2.5664°°** 0.1185°*** 0.2598"*** 0.1605°*** 0.1841°*** 
(0.0046 0.0080) (0.0669) 0.0042) 0.0086) 0.0094) 0.0091 ) 
Student slope T21 0.0036"* 0.0034 0.0520°** 0.0011 —0.0065** 0.0035 0.0009 
(0.0014 0.0019) (0.0157) 0.0012) 0.0023) 0.0027) 0.0027) 
Residual o 0.1097°*** 0.1392°*** 0.7530°°** 0.0954**** 0.1858"*** 0.3325°*** 0.2811°*** 
(0.0031 0.0042) (0.0255) 0.0027) 0.0056) 0.0087) 0.0176) 
Hedges' g Time x Condition g 0.053 —0.040 —0.049 0.003 0.027 —0.038 0.040 
p-Values Time x Condition p 0.792 0.844 0.217 0.739 0.944 0.365 0.989 


Note. Table entries show parameter estimates with standard errors in parentheses except for intraclass correlations (ICCs), Hedges’ g values, and p-values. Tests of 
fixed effects (first four rows) used 39 df to account for the school as the unit of analysis. ICC = intraclass correlation coefficient. ICCs calculated as proportion of 
variance in change over time due to students within the same school (T3; / [T21 + T31]). 

** p< 0.01. 

“py < 0.001. 
“ey < 0.0001. 


The Time x Condition effect appeared to be moderated by the teacher-reported student risk score for student-reported paren- 
tal monitoring (p = 0.025), emotional problems (p = 0.029), and school participation (p = 0.015); state-reported math score 
(p = 0.049); and parent-reported negative school contacts (p < 0.001). Only the latter moderation effect for parent-reported neg- 
ative school contacts met our criterion for statistical significance for a single test given the analysis of multiple measures. To help 
interpret results for the continuous moderators, Fig. 5 provides graphs of condition effects by student risk scores. 

Fig. 5, Panel A, shows the moderation results for student risk score on student report of parental monitoring. Students with 
lower risk scores (left side of Fig. 5, Panel A) did not differ by condition on parental monitoring, which is apparent from the 
mean difference (center, heavier line) of about zero and 95% confidence bounds (outer, lighter lines) that include zero. Moving 
from left to right, the 95% confidence bounds includes zero for scores from 6 to about 15 and excludes zero for student risk scores 
of about 15 and higher, implying a statistically significant difference between conditions for students who had risk scores >15. It is 
important to interpret the moderation effects in the context of the distribution of student risk scores. At least 85% of the students 
had a risk score of 15 or less, so we estimate that the difference between conditions is statistically significant for the students at 


Table 7 
Results from mixed-model Time x Condition analysis of condition effects on academic and teacher-reported outcomes. 
Effect or statistic Math score Reading score Days absent School readiness 
Fixed effects Intercept Boo 221.99"*** (0.7096) 223.85°*** (0.5735) 8.5934"*** (0.3383) 4,4425**** (0.0975) 
Condition Bor 0.3184 (0.9929) — 0.0864 (0.8036) 0.2084 (0.4730) —0.1483 (0.1383) 
Time Bo2 5.1781**** (0.2490) 3.7132"""" (0.0749) 0.3768 (0.2233) — 0.0832 (0.0589) 
Time x Condition Bos —0.1784 (0.3477) 0.1432 (0.1028) — 0.0294 (0.3126) 0.0819 (0.0842) 
Variances School intercept T30 8.9515°*** (2.2424) 5.7204**** (1.4306) 1.8842*** (0.5364) 0.0940** (0.0395) 
School slope 731 1.0666"** (0.2879) 0.8857**** (0.2309) 0.0277** (0.0118) 
Student intercept 720 162.32°*™* (2.4452) 127.57°*** (1.9429) 43.6733°°** (0.8589) 0.3236"*** (0.0473) 
Student slope T21 1.0850** (0.3656) 3.6818**** (0.2302) 0.0037 (0.0122) 
Residual oO 33.9580°°** (0.5526) 34.7117°**™* (0.4306) 20.7805°*** (0.3193) 0.3348"*** (0.0361) 
Hedges' g Time x Condition g —0.017 0.033 0.014 0.095 
p-Values Time x Condition p 0.611 0.172 0.926 0.337 


Note. School readiness was rated by teachers; all other measures provided by the Oregon Department of Education. Table entries show parameter estimates with 
standard errors in parentheses except for intraclass correlations (ICCs), Hedges' g values, and p-values. Tests of fixed effects (first four rows) used 39 df to account 
for the school as the unit of analysis. ICC = intraclass correlation coefficient. ICCs calculated as proportion of variance in change over time due to students/teachers 
within the same school (T31 / [T21 + T31]). Due to challenges with model convergence, the school- and student-level slope variances were fixed to nonnegative 
values for the reading score dependent variable. 

* p< 0.01. 

*** py < 0.001. 
*e*  < 0.0001. 
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Fig. 5. Differences between conditions on four outcomes plotted by teacher-reported student risk of behavior problems. The vertical axis shows the difference be- 
tween conditions (I — C) on gains in each scale across the range of the student scores; a gain of zero represents no difference between conditions. The heavy line 
depicts the mean difference estimate. The two thin, outer lines show the 95% confidence interval around the mean estimate. 


highest risk who comprise about 15% of the student sample. Fig. 5, Panel B, shows significantly lower student-reported emotional 
problems for PFS students with a risk score of about 18 or greater, and Panel C shows significantly greater school participation for 
student with a risk score of about 18 or greater. A risk score of 18 includes only the students most at risk. As shown in Panel D, 5% 
of the student sample shows significantly lower parent-reported negative school contacts for PFS students at a risk score of 12.0 
and greater, approximately 25% of the student sample. The effect for state-reported math score moderated by student risk is not 
shown in Fig. 5 because the 95% confidence interval included zero across the full range of risk scores. That is, although math 
scores were moderated by risk status, with a small negative difference for students at low risk and small positive difference for 
students at high risk, the confidence bounds included zero, implying no statistically significant difference between conditions, 
at all levels of risk. 

The Time x Condition effect was not moderated by student gender or minority status for any study outcomes except possibly 
student-reported positive school structure (p = 0.031), which did not meet criterion with multiple tests. The interaction sug- 
gested positive change for males in PFS schools compared with males in control schools and negative change for females com- 
pared with those in control schools, but an examination of subgroups produced nonsignificant Time x Condition effects for 
both females (t = 0.34, p = 0.738) and males (t = — 1.34, p = 0.264). 

The Time x Condition effect was moderated by cohort for student-reported conduct problems (p = 0.003) and emotional 
problems (p = 0.015), math score (p < 0.001), reading score (p = 0.023), and number of days absent (p = 0.035). With our cri- 
terion (a < 0.001) for multiple tests, cohort moderated only math scores; we report all six results to allow for the interpretation 
of patterns. Cohort 2 showed differences between conditions in growth on student-reported conduct problems (t = — 2.26, p = 
0.030) and state-reported math scores (tf = — 2.08, p = 0.044), whereas no significant changes on these measures were found for 
Cohort 1. For conduct problems, we found an increasing slope for Cohort 2 students in control schools (t = 3.39, p = 0.002) but 
no change across time for Cohort 2 students in intervention schools or for Cohort 1 students in either condition; slopes also dif- 
fered significantly between cohorts within only control schools (t = 3.79, p < 0.001). The pattern was similar, yet less extreme, 
for emotional problems, with a difference in slopes between cohorts within only control schools (t = 2.51, p = 0.016). Math 
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scores increased at a slightly higher rate for Cohort 2 students in control schools than for either Cohort 2 students in intervention 
schools, noted above, or Cohort 1 students in control schools (t = 4.66, p < 0.001). Finally, no statistically significant findings were 
detected for either study cohort for state-reported reading score or days absent. Tests of moderation by cohort may have been a 
result of implementation, in that implementation may have improved over time within each school. The two cohorts, however, 
differed in three other important ways: (a) the grade at which they were first exposed to the intervention, seventh grade for Co- 
hort 1 versus sixth grade for Cohort 2; (b) the number of years of exposure, 2 years for Cohort 1 versus 3 for Cohort 2; and (c) the 
availability of pretest assessment data, with Cohort 2 providing data only after implementation. Moreover, the effects were mixed, 
with improved conduct and emotional problems for Cohort 2 students in the PFS condition but worse math scores in the PFS con- 
dition for those same students. 


3.4.3. Exploratory analyses 

The analyses described previously tested measures and moderators selected a priori as potential targets of the PSF interven- 
tion. Upon finding limited support for the effects of PFS on outcomes selected before we began the analysis, we tested seven ad- 
ditional measures: student reports of parent involvement, depression, problems with other students, deviant peers, prosocial 
behavior, coping, and perceived future career. The main effects were statistically significant for coping but not for any of the 
other student measures or the moderation tests. We also tested universal PFS implementation fidelity as a moderator. PFS imple- 
mentation fidelity moderated condition effects for parent-reported parental monitoring and positive school contacts. In both cases, 
the statistically significant condition effects appeared between schools with the lowest fidelity scores, and parents of control 
school students reported lower scores than did parents of intervention school students. The exploratory analyses added 61 new 
tests: 42 tests of seven exploratory measures for efficacy and moderation (five moderators) plus 19 tests of a priori measures 
with the exploratory moderator. Because only three of the 61 additional tests (4.9%) were statistically significant, we assumed 
they were likely Type I errors. 


4. Discussion 


The purpose of this study was to examine the effectiveness of the PFS model when implemented by school staff under routine 
conditions. As described by Dishion (2011), the intervention was designed to fit within schools that had adopted schoolwide PBIS. 
Systemic concatenation was used to align the core principles of PBIS and PFS, as shown in Fig. 1. The effectiveness-implementa- 
tion hybrid design enabled us to examine the hypothesized intervention outcomes on the basis of the conceptual model presented 
in Fig. 2. It also enabled us to examine implementation fidelity and the factors that may have affected the implementation of the 
PFS tiered supports that emerged from the process evaluation and interviews with school staff. 


4.1. Effectiveness of PFS 


This effectiveness evaluation addressed two primary research questions: Did students in schools provided with PFS improve on 
outcomes when compared with students in control schools, and did the level of risk moderate intervention effects? With respect 
to the first question, students reported improvement in their peers’ positive involvement in school emerged as the only potential- 
ly significant Time x Condition effect (p = 0.031, g = 0.10). Given the number of measures tested, however, this difference did 
not meet our criterion for statistical significance, nor could the single result constitute a pattern. Although the results were disap- 
pointing with respect to the overall impact of the PFS model on parental school involvement, student engagement in school, and 
student social and emotional adjustment, the prior research on components of the PFS model had demonstrated the most success 
with students at risk for problem behavior (e.g., Stormshak et al., 2009). Because the evaluation of the PFS model occurred at the 
school level, the sample included a substantial portion of students with well-managed families, normal social and emotional ad- 
justment, and positive school engagement. Hence, the putative intervention mechanisms may not have had a measurable impact 
on such students and families. 

The investigation of moderation effects demonstrated that among the top 15% to 25% of students at risk for behavior problems, 
as reported by their teachers, students in intervention schools outperformed those in control schools on parent-reported negative 
school contacts and student-reported parental monitoring, emotional problems, and school participation. Fig. 4 shows that as 
teacher-reported behavioral risk increased for students, the difference between the intervention and control conditions improved 
for each of these measures. These results are consistent with previous research that suggests risk behavior leads to greater parent 
engagement, and subsequently, increased effectiveness of parenting interventions over time (Shelleby & Shaw, 2014). Moderation 
effects for only one dependent variable, however, met our criterion for statistical significance of a single test (a < 0.003): parent- 
reported negative school contacts. Similarly, the Benjamini-Hochberg (BH) correction (Benjamini & Hochberg, 1995) for the mul- 
tiple tests, recommended by the What Works Clearinghouse (2014; see also Gottfredson et al., 2015), also suggested that student 
risk only moderated the effect of condition on parent-reported negative school contacts (BH-corrected p = 0.006). The BH- 
corrected p-values were 0.139 each for risk-moderated student reports of parental monitoring, emotional problems, and school 
engagement. These results were nonetheless consistent with the conceptual model presented in Fig. 2 with respect to putative 
treatment mechanisms (parental monitoring) and proximal outcomes of student adjustment (emotional problems, negative 
school contacts). Prior research had shown effects on emotional problems and school engagement, for example, with CACE 
models that test effects for those families that engaged in the FCU (e.g., Fosco et al., 2016), although our results did not replicate 
effects on all measures (e.g., substance use, family conflict, antisocial behavior). In our study, schools may not have provided PFS 
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to a sufficiently large number of behaviorally at-risk students (see the implementation subsection). In such a case, the tests of 
moderation by risk may have underestimated the potential of PFS had schools implemented its components more consistently 
and broadly. Hence, although the effectiveness results were not robust, the pattern of the findings was partially consistent with 
findings from previous research on the PFS components and conceptual framework. 

The evaluation also addressed several secondary research questions captured by tests of moderation by cohort, gender, and 
minority status. Cohort moderation yielded mixed results. Two tests suggested differences between Cohort 2 students in PFS 
schools and control schools on conduct and emotional problems. Increased problems among students in control schools, however, 
appeared to drive the differences between conditions for Cohort 2 rather than a decrease in conduct and emotional problems 
among students in intervention schools. Conduct and emotional problems among intervention students in Cohort 2 were similar 
to those reported by students in Cohort 1 in either the intervention or control condition. Another test suggested decreased math 
scores for the same students, but this, too, appeared (paradoxically) to lead to larger increases in math scores among Cohort 2 
students in the control schools than among the other three Cohort x Condition groups. Two other moderation tests, those for 
reading score and days absent, produced equivocal results, with no statistically significant difference between conditions for either 
cohort. Because of the differences between cohorts in implementation, age at onset, exposure, and data available only at posttest, 
these effects are difficult to interpret. 


4.2. Implementation 


The comparison of PFS implementation between intervention and control schools yielded large differences for the universal 
components of PFS and resources available to parents (g = 1.07 and 1.65, respectively), indicating that the intervention schools 
were able to implement these components effectively. For implementation fidelity for the selected or indicated levels, we found 
smaller differences between conditions than expected (g = 0.32). As described in previous reports (Dishion et al., 2016; Fosco et 
al., 2014; Stormshak et al., 2016), several challenges to implementation emerged from the implementation process evaluation and 
interviews conducted with the PFS school staff. These offer potential hypotheses that might help explain the weak fidelity of im- 
plementation for the selected and indicated components, as described next. 

The implementation of PFS occurred during a time of economic recession and financial hardship in the state of Oregon. As a 
result, schools were limited in a number of ways. At the beginning of the study, many schools suffered significant cuts in funding, 
which limited resources available to implement PFS. As the study progressed, the state of Oregon reduced school funding by about 
10% each year (Oregon State Legislature, 2013). Schools were required to cut staff, implement costly teacher and administrator 
evaluations, cut services, and reallocate space to accommodate growing student populations (Stormshak et al., 2016). Oregon 
schools also had one of the lowest high school graduation rates in the country (68%), which placed academic supports at high 
priority, and yet one of the highest student-teacher ratios (U.S. Department of Education, 2012). Schools participating in this pro- 
ject received funding to support the development of FRCs and offer services such as the FCU at a level similar to that offered by 
some state initiatives (e.g., Chaparro et al., 2012). These funds, however, were insufficient to cover costs of all PFS activities, and 
we were unable to direct those funds exclusively toward PFS activities. Moreover, principal turnover in more than half of the in- 
tervention schools (12 of 21) limited administrator support, critical for successful implementation of innovative approaches to be- 
havioral or academic supports within their schools. In this context, implementing PFS was particularly challenging in that ideal 
implementation would have required staff training, space for FRCs, parent screeners to inform about their students and home en- 
vironments, and increased staff time to work with parents and families. 

Given that many school administrators and staff members saw the implementation of PFS as yet one more program, we had 
expected that the integration within a tiered behavior support system such as PBIS would minimize the additional burden and 
allow for more efficient provision of family management supports. About 60% of participating schools achieved acceptable imple- 
mentation of schoolwide PBIS, but in several, the schoolwide components of PBIS were not fully in place. Some of the challenges 
with PFS implementation stemmed from limited schoolwide PBIS implementation, which required time from PFS interventionists 
to shore up schoolwide systems and practices. It is also possible that the successful integration of PFS within PBIS systems, an as- 
sumption on which our study rested, was too resource intensive or otherwise untenable for schools. Future research should delve 
into the implementation dynamics of merging two independent and ostensibly compatible systems in order to more systemati- 
cally uncover the synergies and barriers within school systems. 

Improving fidelity likely requires improvements in training, simplification of the model, or reduced demands on the schools. 
Dishion et al. (2016) have discussed how the PFS model has been simplified and how online training and web-based supports 
might alleviate some implementation challenges. Our study included schools that had implemented PBIS, but only at the univer- 
sal, schoolwide level and not necessarily to criterion. Schools that meet high fidelity standards for all three PBIS tiers, however, 
would most likely offer better targets for PFS implementation. As recommended by implementation science frameworks (e.g., 
Meyers, Durlak, & Wandersman, 2012), conducting a capacity or readiness assessment would have helped gauge whether the 
schools had the necessary structure and supports in place for successful implementation of the PFS model. Furthermore, given 
the small differences between study conditions in implementation fidelity reported for Tiers 2 and 3 of PFS, enhanced training 
procedures and technical assistance may be required to improve both implementation and effectiveness outcomes. In some 
PBIS implementations, for example, trainers or coaches attend monthly school meetings on student behavior for 2 years to devel- 
op a solid core of in-school expertise that can sustain implementation in light of key staff member or administrator turnover 
(Horner et al., 2009). 
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Finding the optimal blend of implementation fidelity and adaptation (Durlak & DuPre, 2008) will require additional research. 
Striving to attain perfect fidelity to implementation may be unrealistic, but if future work could disentangle the intervention's the- 
ory of change from more pragmatic aspects, it may engender adaptations that improve effectiveness. Sundell, Ferrer-Wreder, and 
Fraser (2014) discuss the unanticipated influences of the contexts in which interventions may be placed. Although all schools 
work toward improving academic achievement, they frequently have different missions or emphases, vary in their social and be- 
havioral norms and language, and include diverse populations of students and families in terms of not only race and ethnicity, but 
culture, politics, religion, and other factors that may influence the adoption and implementation of PFS practices. Finding ways to 
build on the culture and language of individual schools may help achieve greater buy-in and adherence to the model. Schools also 
need measures of implementation fidelity at each tier that assess the key theoretical components. A comparison with schoolwide 
PBIS may be instructive, because implementation has shifted from individual schools to entire districts to establish district-level 
trainers and coaches who understand both the dynamics of their school and the fundamentals of the behavior support model. 
The economic analysis of schoolwide PBIS programs by Blonigen et al. (2008) suggests that districts should not only train staff 
members at the school and district level and collect intensive fidelity measures, but also create a district-level leadership team 
to coordinate services and contract for an external evaluation of implementation. 

Finally, the real-world implementation of PFS, required by this effectiveness trial, necessarily introduced more noise in the data 
than efficacy trials because we could not control the implementation of PFS activities conducted by school staff members or col- 
lect important information about them. Implementation data about the families who received PFS supports from the schools were 
not available to the program evaluators. Hence, it was not possible to evaluate the specific effects of each of the PFS tiered levels 
of support or conduct analyses on only families that engaged in PFS components. Although the moderation analyses provide some 
evidence that at-risk students benefited more from the PFS intervention than did lower risk students, the effects associated with 
the specific levels of tiered support cannot be determined from the current trial. Implementation data for Check-In/Check-Out also 
were unavailable, leaving open questions about whether the training and consultation activities changed schools’ use of the inter- 
vention. Nor could we track or control for all other interventions or supports offered by schools to students or their families. Stu- 
dents in every school were placed on behavior support plans or individualized education programs, or received other 
interventions or services (e.g., mental health counseling). Cataloging all these interventions, in addition to all aspects of PBIS 
and PFS, while cost prohibitive, would have allowed for a more sensitive evaluation. 


4.3. Limitations 


Several limitations of this study warrant discussion. The measurement and sampling procedures raised additional limitations. 
The score reliabilities of a subset of the outcome measures were marginal, such as conduct problems (a = 0.63) and emotional 
problems (a = 0.69). These reliability levels were consistent with other reports of the measure (e.g., Kovacs & Sharp, 2014). Al- 
though “there is no specific criterion of acceptability” for reliability values (Rosenthal & Rosnow, 2008), the low score reliabilities 
may have reduced the sensitivity to intervention effects. Similarly, measures of positive and negative school contact by parents 
with unknown psychometric qualities. This is notable given the only statistically significant result, after controlling for multiple 
tests, was a reduction to parents’ reports of negative school contacts and for only a subset of parents. 

The low rates of parent survey completion raises questions about the representativeness of the respondent sample and the po- 
tential for biases. Inherent to most longitudinal school-based research, different teachers reported on the students over time, 
which could have shifted the longitudinal trajectories of student functioning across the assessment periods. Student surveys 
may also have included biases associated with self-report. We aimed to mitigate these challenges by including multiple sources 
of information, including objective measures of academic outcomes and attendance, but it is noteworthy that the only statistically 
significant finding involved parents’ self-report of negative school contacts. Student attrition rates were also rather substantial and 
may have had an influence on the study findings. We implemented analysis procedures that minimize bias associated with attri- 
tion, but because students at highest risk for behavioral and academic problems tend to be the most mobile, and because many 
PFS components target high-risk students and their families, student transience may have concealed or limited intervention ef- 
fects among at-risk youths. 

The discussion about challenges with implementation has been inferred from interviews with school administrators, anecdotal 
reports from research personnel, and from implementation fidelity data. Any conclusions have yet to be verified experimentally 
and should therefore remain tentative until further support can be obtained. In addition, given the limited support for PFS 
from this study, we cannot claim that the effects were solely associated with PSF and not schoolwide PBIS or other activities of 
the schools. 


4.4. Conclusion 


The PBIS framework allowed for the systematic concatenation of PFS with PBIS (Dishion, 2011), and it may continue to serve 
as a framework for further research on implementation of systems to improve parent management and other parent supports and 
increase parent-school involvement. Although the findings derived from this research were weaker than expected across the ef- 
fectiveness and implementation outcomes, this study contributes to the field of prevention science by furthering our understand- 
ing of implementing effective prevention and intervention into schools. Preventing problem behavior and supporting youth 
success is a complex enterprise and requires a commitment of financial resources on the part of state systems. A change in our 
policies, structure of schools, and resource distribution is needed to successfully prevent problems at the school level. Prior 
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research demonstrates that prevention is a cost-effective means of reducing later problem behavior, substance use, and risk be- 
havior in adults. In fact, cost-effectiveness research suggests than an investment of one dollar returns more than five dollars 
when prevention is successful (Kuklinski, Briney, Hawkins, & Catalano, 2012). Our efforts to continue to support schools in imple- 
mentation of evidence-based programs to reduce problem behavior are critical for changing this financial model and for 
restructuring education to support the success of all students. 
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